A semi-supervised deep-learning approach for automatic crystal structure classification

A semi-supervised model to predict crystal structures from powder neutron diffraction patterns has been developed. The models have higher accuracies than current approaches while covering more space groups.


Introduction
The first step towards understanding the properties of a crystalline material at a microscopic level is identifying the crystal structure. However, this is nontrivial. The first part of crystal structure determination is indexing. There are several programs which can be used, such as DICVOL06 (Boultif & Louë r, 1991), TOPAS (Coelho, 2018), GSAS-II (Toby & Von Dreele, 2013) or N-TREOR (Werner et al., 1985;Altomare et al., 2000). These programs output a set of space groups and lattice parameters that could represent the crystal. Using Le Bail (Le Bail et al., 1988) and Pawley (Pawley, 1981) refinements, the space group that fits the diffraction pattern the best can be identified. Rietveld (Rietveld, 1967(Rietveld, , 1969) refinement can then be applied to profile the lattice parameters and check the space group. In the presence of impurity phases, this approach becomes more expensive as peaks must be selected manually or tolerance levels must be tuned to discard a certain number of peaks.
One of the approaches for identifying the positions of atoms in a crystal is the charge-flipping algorithm (CFA) (Oszlá nyi & Sü to , 2008;Palatinus, 2013;Baerlocher et al., 2007). CFA is an iterative approach that relies on fast Fourier transforms to determine the crystal structure of a material (Nussbaumer, 1981;Palatinus & Chapuis, 2007). For CFA, the unit cell and Bravais lattice have to be known, that is, we must have already been successful with some degree of indexing. CFA also cannot handle impurity phases, which are prevalent in many real-world samples.
Data science methods are being used increasingly in materials development (Balachandran, 2020;Vandermause et al., 2020;Reyes & Maruyama, 2019;Karigerasi et al., 2018). An example of this is the use of supervised neural networks (NNs) to analyze diffraction patterns. Supervised learning is an approach that seeks to learn a functional mapping between data and their labels. The benefit of NNs is that they, unlike CFA, do not require additional parameters, such as the Bravais lattice or lattice parameters. Although some approaches use NNs to aid in the Rietveld refinement (Ozaki et al., 2020;Chang et al., 2020;Schmidt et al., 2019;Schleder et al., 2019), others use NNs to classify diffraction patterns on the basis of the crystal structure. These classifiers can be trained to identify impurity phases and can be tailored towards specific detectors or parameters. For example, Ryu et al. (2019) trained an NN to classify the diffraction patterns of crystals that had defects. Liu et al. (2019) used the pair-density function with powder neutron diffraction data for space-group classification. Ziletti et al. (2018) used a convolutional neural network to classify simulated single-crystal diffraction X-ray image data into eight space groups.
A number of studies represent powder diffraction patterns as 2D images. However, the information is inherently one dimensional. Previous groups probably used the image approach to take easy advantage of trained models developed by the machine-learning community. Unfortunately, this could introduce more complexities to the model. Garcia-Cardona et al. (2019) carried out one of the only studies to examine neutron scattering data and used a 1D approach with simulated powder diffraction data to both differentiate perovskites into five crystal systems and tune the lattice parameters using regression. This study only looked at a small subset of crystals.
A significant challenge with NNs is that they struggle to generalize to new data sets. Most models that predict the space group of a material use less than 100 space groups in their training data set, which limits their application to new diffraction patterns. However, large labeled diffraction data sets are often rare, as labeling them is an expensive task. For this reason, we use a semi-supervised model, which takes advantage of both labeled and unlabeled data during training (Odena, 2016;Zhu & Goldberg, 2009;Kingma et al., 2014;Kipf & Welling, 2016). We employ a generative network that can extract features from the unlabeled data distribution and match these features with the corresponding crystal structure. This allows semi-supervised learning to be used on more data sets, especially ones where labels are not available.
In this study, we propose a 1D semi-supervised model for Bravais lattice and space-group classification using powder neutron diffraction data. Our NNs are trained with data spanning 144 space groups and 14 Bravais lattices. The models used in this study are freely available and can be downloaded (Lolla & Liang, 2021).

Data
To test our approach under conditions where we know the correct answer, we worked with simulated data sets. Our data were taken from the Inorganic Crystal Structure Database (ICSD; https://www.psds.ac.uk/icsd), which contains structural information about more than 210 000 crystals (Bergerhoff et al., 1983). A total of 138 362 diffraction patterns were simulated using TOPAS (Coelho, 2018). For the Bravais lattices, we combine the rhombohedral and tetragonal classes for a total of 14 classes with 'F', 'I', 'P' and 'C' representing the facecentered, body-centered, primitive and base-centered lattices, respectively. We note that there is an inherent class imbalance in the ICSD, as shown in Fig. 1. The most prevalent classes in this data set were the primitive hexagonal, the face-centered cubic and the primitive orthorhombic lattices. The least represented lattices were the face-centered orthorhombic and body-centered orthorhombic lattices.
For the space groups, we used 136 454 of the simulated diffraction patterns. We used only the space groups that had more than 50 diffraction patterns, leaving us with 144 out of the 225 space groups present in the ICSD. The most frequent space group is No. 62 (Pnma), which is orthorhombic, and accounts for the disproportionately large number of orthorhombic (P) diffraction patterns in the ICSD data set. A complete list of the 144 space groups used is shown in Table 1.

Figure 1
The (a) Bravais lattice and (b) space-group class distribution of the simulated data.
A complete list of the ICSD IDs used in this study can be found in the GitHub repository at the URL https:// github.com/usnistgov/semi-supervised-neutron (Lolla & Liang, 2021).
In this study, we use a 1D approach rather than the traditional 2D image approach. Our data set consists of diffraction patterns of powders. Examples of the 1D diffraction patterns are shown in Fig. 2. To normalize these diffraction patterns, we divided all intensities in each diffraction pattern by the maximum intensity. This ensures that the new maximum intensity is equal to 1 and the minimum is equal to 0.

Models
We use two approaches to classify the diffraction patterns: a supervised approach using convolutional neural networks (CNNs) and a semi-supervised approach using a semisupervised generative adversarial network (SGAN).

Supervised model
We used a 1D ResNet-18, a residual network (He et al., 2016), model to identify the crystal structure of the diffraction patterns.

Semi-supervised model
We also used an SGAN (Odena, 2016;Goodfellow et al., 2020;Salimans et al., 2016). The SGAN consists of two models: a Generator and a Discriminator. The Generator tries to fool the Discriminator with fake diffraction patterns, while the Discriminator aims to differentiate between real and fake diffraction patterns. The Discriminator also classifies the real labeled data into the corresponding crystal structure class.
2.4.1. Generator. The purpose of the Generator is to sample the latent space, a high-dimensional feature space, to generate realistic diffraction patterns. The inputs to the Generator were sampled from a random normal distribution with a mean of 0 and a standard deviation of 1. The Generator consists of 1D Convolutional Transpose layers, 1D Batch Normalization layers and a Leaky rectified linear unit (ReLU) activation function. The Convolutional Transpose layers are used to upsample the data (Radford et al., 2015;Dumoulin & Visin, 2016). The Batch Normalization layers standardize the output of each layer, which reduces error when the model tries to generalize to new inputs (Ioffe & Szegedy, 2015) and has also been shown to reduce mode collapse, a major problem in GANs (Radford et al., 2015). Mode collapse occurs when the Generator only produces a few distinct diffraction patterns despite the latent space input. The Leaky ReLU (Xu et al., 2015) activation with = 0.2 is used rather than ReLU to reduce the vanishing gradients problem (Radford et al., 2015). Graphs of the ReLU and the Leaky ReLU activation functions are shown in Fig. 4. For negative values, the derivative for the Leaky ReLU function is equal to , but for the ReLU function, it is equal to 0. By having a nonzero derivative for all values, the Leaky ReLU is used to combat the sparse gradient problem that occurs while training GANs. Due to our normalization method, which was dividing all values in a diffraction pattern by the maximum intensity, the Discriminator's inputs were in the range from 0 to 1. For this reason, a sigmoid activation was applied to the last layer of our  2.4.2. Discriminator. The Discriminator has two objectives: to differentiate between real and generated data, and to classify the real data into the correct class. To do this, we used the same 1D ResNet-18 model described in Section 2.3, but an activation function to the last fully connected layer, as proposed by Salimans et al. (2016). This activation function is shown in equation (1) and is a version of the softmax activation: In this equation, l k ðxÞ represents the logit for class k with data x. By doing this, we eliminate the need for a second output layer and instead use only the logits from the classification layer. By applying this activation function, diffraction patterns with larger logits, which signify more confident predictions, will be classified as 'real', whereas diffraction patterns with smaller logits will be classified as 'fake'. This encourages the Discriminator to be more confident in its predictions, which sharpens the decision boundary between classes. The Discriminator's architecture is shown in Fig. 6. While training the Discriminator, there are two modes: supervised and unsupervised. During unsupervised training, the Discriminator acts the same way it would in a regular GAN as it tries to determine that the generated diffraction patterns are fake and the data drawn from the unsupervised set is real. In the supervised mode, the Discriminator is trained to predict the class label for real samples. Training in the unsupervised mode can help the Discriminator extract features from the data, and training on the supervised data will allow the Discriminator to use those extracted features for classification.
2.4.3. Loss functions and objective functions. The modified min-max loss proposed by Goodfellow et al. (2020) was used for the adversarial loss between the networks. The objective function that the Generator tries to maximize is shown in equation (2): g represents the parameters in the Generator and z ðiÞ represents the random values in the latent space. Gðz ðiÞ Þ is the generated diffraction pattern from the Generator and À logfD½Gðz ðiÞ Þg is the probability that the Discriminator predicts that the generated pattern is real.
For the Discriminator, the objective function to be maximized is shown in equation (3)  The architecture for the Generator. The blue, green, orange and red boxes represent 1D Convolutional Transpose layers, Leaky ReLU activations, Batch Normalization layers and sigmoid activations, respectively. 'nc' and 'ks' in the blue boxes represent the number of channels and the kernel size, respectively, for each 1D Convolutional Transpose layer.

Figure 6
The model architecture for the ResNet-18 Discriminator. The orange, purple, green, red, gray, blue and yellow layers represent 1D Convolutional layers, Dropout layers, Layer Normalization layers, GELU activation functions, 1D Max Pooling layers, 1D Adaptive Max Pooling layers and Fully Connected layers, respectively. The white circles with '+' signs in them represent the addition of two layers. In part (b), each ResNet block was repeated twice. 'nc' and 'ks' represent the number of channels and the kernel stride for each convolutional layer, respectively.
d represents the parameters in the Discriminator, x ðiÞ is the unsupervised real data, and maximizing Dðx ðiÞ Þ implies that the model can identify real data. Like the Generator's objective function, z ðiÞ represents the random values in the latent space and Gðz ðiÞ Þ is the generated diffraction pattern from the latent space. Increasing the value of 1 À logfD½Gðz ðiÞ Þg shows that the Discriminator can determine that the generated patterns are fake. Equation (3) also includes the categorical cross entropy loss, which is shown in the term P C i t i logðs i Þ. Here, C represents the number of classes, t i shows whether the ith class is the label of the diffraction pattern and s i is the Discriminator's prediction.
2.4.4. Training details. Fig. 7 shows the training pipeline used in the SGAN. During SGAN training, the Discriminator has three inputs: a generated sequence from the Generator, a powder diffraction pattern that is labeled with either the space group or the Bravais lattice, and an unlabeled powder diffraction pattern that does not include the crystal structure. We train our SGAN using four different amounts of labeled training data. In all scenarios, we randomly select 10% of the data as testing data, which is distinct from the labeled training data and the unlabeled training data. In the first scenario, we use 5% of the data as labeled training data and 85% as unlabeled training data. In the second, we use 10% of the data as labeled training data and 80% as unlabeled training data. In the third, we use 25% of the data as labeled training data and 65% of the data as unlabeled training data, and finally we use 50% of the data as labeled training data with 40% of the data as unlabeled training data. We also train our supervised ResNet with the same 5, 10, 25 and 50% of the data to compare the accuracy of the SGAN with that of the purely supervised approach.
To train a supervised classifier, we use only the percentage of labeled training data. The model uses a powder diffraction pattern as input and aims to differentiate between the various crystal structure classes. Table 2 shows the hyperparameters used in the ResNet and the SGAN.
We used PYTORCH (Paszke et al., 2017) as a deep-learning framework. To accelerate training, each model was trained on eight NVIDIA Tesla V100 Tensor Cores.

Supervised model
Our supervised ResNet trained on 90% of the data set had an accuracy of 88%. The confusion matrix for the Bravais lattice model is shown in Fig. 8. By plotting the predicted Bravais lattice against the actual Bravais lattice, the confusion matrix provides more information about the sets of classes that the network misclassified. If the model had a perfect testing accuracy, the values along the principal diagonal would sum to 100% as the network would have classified every diffraction pattern correctly. Again, there is a clear imbalance in the sampled ICSD data set, with orthorhombic (F) and orthorhombic (I) having the least samples. From the confusion matrix, we can see that, despite the fact that orthorhombic (P) is the most prevalent class, the model misclassifies some of these as monoclinic (P) crystals. The network also has trouble differentiating between triclinic (P) and monoclinic (P) diffraction patterns, as both of these classes have low symmetries, agreeing with previous studies (Garcia-Cardona et al., 2019). Similarly to Suzuki et al. (2020), we believe that this result was caused by undersampling the triclinic crystals.
For the space-group identification, our model had a top-1 accuracy of 80.6% and a top-5 accuracy of 90.27% across all 144 space groups. We trained our model on all 230 space groups and found that the model had a top-1 accuracy of 74% and a top-5 accuracy of 85%. We decided to investigate further the model on only the 144 most prevalent space groups within  The architecture for the semi-supervised GAN. Table 3 Comparison with other supervised space-group classifiers. the data set, due to a major class imbalance. Some space groups had less than 50 diffraction patterns, less than 0.03% of our data set. Accuracy was measured by dividing the number of correctly classified diffraction patterns in the testing set by the total number of patterns in the testing set. Top-5 accuracy is the percentage of samples for which the actual space group was one of the model's top five predictions. This outperforms most current models of which we are aware. Liu et al. (2019) used machine learning with a pairwise distribution function with a top-1 accuracy of 71% and a top-5 accuracy of 90% across 45 space groups. Tiong et al. (2020) classified X-ray diffraction data into 8, 20, 49 and 72 space groups (Table 3). Their accuracy decreased from 99 to 80% for 8 and 72 space groups, respectively, implying that this accuracy would decrease further if their model was trained on more space groups. Aguiar et al. (2019) had a top-2 accuracy greater than 80% across all space groups, but used a data set consisting of 650 000 diffraction patterns, more than five times the size of the data set used in this study. However, they used a 1D network, suggesting that a 1D approach can lead to more accurate predictions. We note that we did not take advantage of data augmentation.

Semi-supervised model
We compare the accuracy of the SGAN with the accuracy of the supervised model in Table 4. The SGAN consistently outperforms the purely supervised model, showing that the semi-supervised approach has the potential to be more applicable in the real world. A graph comparing the accuracy of the supervised and semi-supervised models is shown in Fig. 9. This graph shows that although the accuracy of the SGAN is impacted by a lack of data, the difference between the accuracy of the SGAN and the accuracy of the supervised model is greatest when only 5% of the data are used.

Conclusion
In this study, we use both CNNs and a semi-supervised GAN to investigate supervised and semi-supervised approaches for crystal structure classification. We demonstrate that SGANs can prove to be more accurate with limited quantities of labeled data for Bravais lattice and space-group classification. Further, we explore a 1D approach rather than a traditional 2D one. Our 1D model is more accurate than 2D image models, which agrees with previous results in the literature. Our semi-supervised model is also more applicable to real data sets which will lack large quantities of labeled data.
In the future, we would like to train the SGAN to identify impurity phases and to test the method on real data sets. Table 4 Comparing the accuracy (%) of the SGAN to the purely supervised approach with different quantities of labeled training data for both Bravais lattice and space-group classification.